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The purpose of the research is to develop and identify the validity of factors affecting higher order 
thinking skills (HOTS) of students. The thinking skills can be divided into three types: analytical, 
critical, and creative thinking. This analysis is done by applying the meta-analytic structural equation 
modeling (MASEM) based on a database of 166 primary empirical studies. The research results assert 
the theories and bring conceptual and empirical clarity to the factors affecting HOTS of students and 
also give readers an understanding of the magnitude and significance of relationships among the 
variables in the model. MASEM results confirm that classroom environment, psychological and 
intellectual characteristics of students have direct effects on HOTS (96.8% explained variance). 
Whereas, the family characteristic had insignificant effects on HOTS but they had indirect effects on 
HOTS through psychological characteristic. Furthermore, we show that the most direct effects on HOTS 
were psychological characteristic, classroom environment and intellectual characteristic, respectively. 
This study provided a holistic view on the relationship of factors affecting HOTS and proposed a 
direction for future research and practice. 

Key words: Higher order thinking skills, meta-analytic structural equation modeling, classroom environment, 
family characteristic, psychological characteristic, intellectual characteristic. 


INTRODUCTION 

Higher Order Thinking Skills (HOTS) is a thinking 
process, which consists of complicated procedures and 
needs to be based on various skills such as analysis, 
synthesis, comparison, inference, interpretation, assess¬ 
ment, and inductive and deductive reasoning to be 


employed to solve unfamiliar problems (Smith, 1992; 
Zohar and Dori, 2003). The characteristics of students 
with HOTS are open-mindedness for risk-taking, curiosity, 
keen on fact discovery, planning and indicating the most 
suitable method, have a systems thinking process, think 
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carefully, use evidence to think rationally and frequent 
self-monitoring (Shari et al., 1993). The students with 
HOTS are able to create new knowledge and make 
appropriate and logical decisions. Information and 
technology advancement greatly influences the current 
society. Consequently, learning management must be 
adapted to the current situation/society and focus on 
improving HOTS of students. 

There are many concepts of HOTS applied to the 
educational development of students and these concepts 
have been studied for years and used for teaching and 
learning in the classroom and the research of factors 
contributing to students’ HOTS development (Noble and 
Powell, 1995; Rajendran, 2001; O’Tuel and Bullard, 2001; 
Marshall , Robert and Horton, 2011; Magno, 2011; 
Fischer et al., 2011; Kondak and Ayden, 2013). Within 
the thinking process literature, there are many factors 
affecting HOTS: classroom environment, family charac¬ 
teristic, psychological characteristic, and intelligence 
(Horan, 2007; Silvia, 2008; Pannells and Claxton, 2008; 
Lim and Smith, 2008; Chini et al., 2009; Pascarella et al., 
2013; Fearon et al., 2013; Lather et al. ., 2014). These 
factors are related and mutually supported. Thus, the 
aforementioned factors should be included in the 
teaching model, which will be of benefit in supporting and 
promoting the development of HOTS. However, despite 
more than a decade of studies in this area and a variety 
of models proposed to explain the factors affecting 
HOTS, the extant factor affecting HOTS literature 
remains as follows; 1) the lack of systematic integration 
among those variables, 2) the researcher can not 
specified all relationships by a theory needed to be 
included in each primary study 3) some relations that are 
inconsistent or contradict one another across studies 
(Montea and Siu, 2002; Brink, 2003; Cheung and Chan, 
2005; Montazemi and Hamed, 2015). It is difficult to draw 
conclusion from these studies. Hence, it is necessary to 
have a summary of research findings of the increased 
studies, variety of concepts on factors affecting HOTS. 
These will be studied further for a clear conclusion and a 
similar direction to get the most benefit from the 
information and to develop the most effective practical 
application. A systematic meta-analysis is likely to help 
us solve these problems. 

For the research methodology of data collection and 
conclusion, the researcher employed a research 
synthesis method to data collection and applied statistical 
procedures to draw conclusion and solutions of the 
problems (Light and Pillemer, 1984). During the first 
stage, the descriptive method, traditional vote-counting 
methods and commutation of p-values are used for 
quantitative synthesis. Later, the meta-analysis is used 
by integrating the effect size to the analysis process to 
acquire better synthesis results and to delete the 
disadvantages of traditional synthesis which give 
subjective results (Kulik and Kulik, 1989). More recently, 


the meta-analytic structural equation modeling approach 
(MASEM) has been developed for advanced statistics 
from more complex variance models which give research 
conclusions in terms of the causal relationship from 
different research. This also resulted in affirming or 
denying the theoretical relationship structure. Moreover, it 
will provide a powerful means for testing broader, richer, 
and more complex theories that are unlikely to be feasibly 
tested in any single primary study (Viswesvaran and 
Ones, 1995; Hunter and Schmitdt, 2004). Additionally, 
the research results indicated a causal relationship model 
of both direct and indirect effects in real situations 
(Bamberg, 2007; Yu and Chiu, 2007). In conclusion, the 
research indicates that the MASEM is more practical and 
informative than the traditional meta analytic method. 

From the problems and major issues mentioned above, 
it is obvious that there has not been conclusion from 
analyzing the factors affecting HOTS by applying 
MASEM in Thailand. The problems could be caused by 
the following; 1) unfamiliarity with the analysis method, 2) 
few applications for education and 3) some factors are 
overlooked. Moreover, some repeated problems are 
caused by education systems and policies, which reflect 
the limitations of research analysis due to the same 
results provided with no differences, thus the result of 
research database is not enough to analyze by MASEM 
(Borenstein et al., 2009). Therefore, the author is aware 
of the advantage and the application of MASEM and 
HOTS for research in human and social sciences. 
Consequently, the educationists use the information to 
determine what factors directly resulted in HOTS 
development and what elements relate to them. The 
outcomes of this analysis will contribute to effective 
development of students’ HOTS. Additionally, it is also 
considered to be an extension of the knowledge 
application of MASEM analysis for the future studies. 


LITERATURE REVIEW 

Higher order thinking skills; HOTS 

The definition of HOTS is the ability and expertise to find 
answers or achieve target goals through various forms of 
thinking processes. It is necessary for students to learn 
and practice this ability in order to acquire answers, to 
make decisions, and to solve problems (Lewis and Smith. 
1993; King et al., n.d.). Educators have an assortment of 
HOTS that include several concepts. Krulik and Rudnick 
(1993) state that HOTS includes 1) recall thinking, 2) 
basic thinking, 3) critical thinking, and 4) creative thinking. 
Byrnes (1996) classifies HOTS into 4 levels; 1) the 
application level, 2) the analysis level, 3) the synthesis 
level, and 4) the evaluation level. Anderson and Krathwohl 
(2001) propose the concepts of Bloom’s Taxonomy 
Revised, and classify cognitive approaches to learning 
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into six levels; 1) remembering, 2) understanding, 3) 
applying, 4) analyzing, 5) evaluating, and 6) creating. 
Based on the national standards of educational 
management and basic curriculum of Thailand, the key of 
these concepts related to HOTS development are the 
main focus for the development of characteristics in 
students’ thinking skills. Moreover, they are the variables 
the author use in this study; 1) Analytical thinking: AnT is 
the ability of individuals to classify objects logically, 
assessing the relationships of certain elements, how they 
contribute, how they relate to each other, how they work, 
and what the most important parts are (Bloom et al., 
1956; Marzano, 2001). 2) Critical Thinkingical Thinking; 
CriT refers to the ability to evaluate and consider things 
by searching for reliable and sufficient information before 
making decisions, solving problems, evaluating situations 
and taking action on any tasks with the most appropriate 
and accurate ways (Ennis, 2002; Black and Black, 2006; 
Ellis, 2009). 3) Creative Thinking; CreT refers to thinking 
competency in using previous knowledge to create new 
knowledge for discovering or innovating new things. This 
often results in more valuable outcomes, which can be used 
or applied to problem solving or effective performance 
(Sternberg and Lubart, 1999; Harvey, 2010). 


Meta-analytic structural equation modeling; MASEM 

Meta-analytic structural equation modeling (MASEM) is 
the most recently developed quantitative synthesis 
technique, which combines two research methodologies. 
Meta - analytic (MA) is the statistical analysis of analysis 
results from individual studies for the purpose of 
integrating the findings in form of effect size. Structural 
equation modeling (SEM) is a technique used to verify or 
test theoretical causal models (Glass, 1976; Hunter and 
Schmidt, 2004; Cheung, 2008). For the first phase, meta¬ 
analysis was synthesized to draw a conclusion of the 
effect size as an index of the direction and magnitude of 
the association between two variables, which includes 
Pearson correlations (r) and standardized mean 
difference (g). In order to conclude the effect size of more 
complex variables, the effect size on a series of 
correlation matrices is used to create a pooled correlation 
matrix, which can then be analyzed using SEM 
(Viswesvaran and Ones, 1995; Shadish, 1996; Cheung 
and Chan, 2005; Hafdahl, 2009). 

Landis (2013) states that there are at least two primary 
approaches that serve as a foundation for integrating MA 
and SEM. 1) The analysis model proposed by 
Viswesvaran and Ones (Colquitt et al., 2000; Earnest et 
al., 2011; Robbins et al., 2009) is applied when no study 
provided full information of all variables indicated in the 
models. 2) The two-stage SEM (TSSEM) proposed by 
Cheung and Chan (2005) is a preferable alternative for 


the author to apply when there is at least one study 
provided full information. In the present paper the author 
conducted MASEM by following a two-stage procedure of 
Viswesvaran and Ones (1995), which was considered to 
be the most suitable method for the data in this study. 
The concepts of analysis consist of two models that have 
continuous processes related to each other: the 
measurement model and the casual model. The five 
steps of the measurement model for theory testing are 1) 
identifying important constructs and relationships, 2) 
identifying different measurements used to operationalize 
each construct, 3) indicating all relating statistics and all 
of their importance in of studies, 4) processing the meta¬ 
analyzing and estimating the real value of the relationship 
of the measurement, 5) using factor analysis to test the 
measurement models. For casual models, there are the 
processes of measurement as following: 6) estimating the 
correlation value between structures from different 
structures, and 7) using path analysis with the estimated 
true value of correlation to test the proposed theories. 


Classroom environment; CIEnv 

The previous studies of classroom environment revealed 
the factors affecting the environment to enhance the 
effective teaching and learning processes are learning 
achievement, desirable characteristics of students, and 
processes of skill development including HOTS (Brown 
and Freeman, 2000; Dorman, 2002; Fisher and Khine, 
2006; Wolf and Fraser, 2008; Galton et al., 2009; 
Pascarella et al., 2013). Even there were results 
indicating that factors concerning classroom environment 
were differed and variety, but from the author synthesis 
the variables of classroom environment affecting HOTS 
can be divided into three factors; 1) Classroom climate; 
CICIi refers to learning environment for both physical 
atmosphere such as tidiness, cleanliness, light, and size, 
and psychological atmosphere such as safety, warmness 
and good relationship, and freedom in expressing ideas 
and feelings (Moos, 1979; Dunn and Dunn, 1992; Brand 
et al., 2003; Ambrose et al., 2010; Wanekezi and Iruloh, 
2012)., 2) Teaching and learning methods; TeM refers to 
principles, methods, patterns, and techniques that 
teachers apply to manage students’ learning and to 
achieve classroom management goals (Jones et al., 
1987; Alberta Learning, 2002)., and 3) Teacher behavior; 
TeB refers to the actions of teachers in classrooms to 
motivate, facilitate, and encourage students for 
performing their efficient works (Dorman, 2009). 


Family characteristic; FaCh 

Family is a basic social unit where parents insert their 
love, cares, values, attitudes, and life experiences for 
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students. Therefore, this factor is considered a foundation 
for every dimension of students’ development as well as 
an influent element affecting students’ learning outcomes 
and thinking skills, which showed the individual 
differences (Jackson, 2003; Wade, 2004; Campbell and 
Gilmore, 2007). Regarding the previous studies, the 
results show that there are two major factors of the family 
characteristic; 1) Democratic parenting style; Dmo refers 
to the method used by parents to take care of their 
children informally, but remain the rules with reasonably 
and democratically acceptances (Baumrind, 1966; 
Maccoby, 1992; Steinberg, 2001)., 2) parental support; Sup 
refers to the assistance, support, encouragement, and 
conveniences provided to children to live and learn 
including the learning environment to enhance students 
to gain new experiences and develop more advance 
skills (Ghate et al., 2000; Patricia et al., 2004). 


Psychological characteristic; PsyCh 

The psychological characteristic refers to the personality 
trait or behavioral characteristic which affects the learning 
strategy and the thinking process of individual to express 
students’ feelings to contribute to their different learning 
and thinking skills (Lahey, 2001; Sternberg and Willium, 
2001; Woolfolk, 2004; Santrock, 2009). The studies show 
that the two major factors of psychological characteristic 
are 1) Attitude toward learning; Atti refers to the student’s 
ability to show satisfaction, and the agreement and 
disagreement toward classroom environment, teachers, 
learning activities, classmates and curriculum (Zimbardo, 
1999; Bernstein et al., 2006)., 2) Achievement motivation; 
Moti refers to students’ willingness, intention, enthusiasm, 
and attempt to achieve learning objectives with high 
performance (McClelland, 1961; Woolfolk, 2004)., and 3) 
Internal locus of control; Loc refers to students’ self- 
awareness competency in working and achieving the 
goals, or even when they fail on their tasks, they keep 
their focus and effort to be successful (Rotter, 1990; 
Stajkovic and Luthans, 2003). 


Intellectual characteristic; IntCh 

According to the literature reviews, the findings show that 
intellectual characteristic also covers intellectual 
competency, solving problems and reasoning to change 
learning behavior, and differences of thinking process 
skills of individuals (Kane et al., 2004; Kim, 2005; Horan, 
2007; Silvia, 2008). The results of synthesis show two 
major factors of intellectual competency, which are 1) 
Intelligence quotient; IQ refers to competency in learning, 
solving problems, and adjusting to new environments and 
problems (Feldman, 1992; Woolfolk, 2004)., 2) Reasoning 
abilities; Reas refers to the ability in transferring previous 


knowledge to new experiments through thinking processes, 
solving problems, and finding relationships of things to make 
decisions based on the current information and problems 
(O’Daffer, 1990). 


Objective 

To develop and assess the validity of a structural equation 
model of factors affecting HOTS through meta-analytic 
structural equation modeling. 


Hypothesis 

The research hypotheses are given in Table 1. The 
theoretical models of the factors affecting HOTS are 
shown in Figure 1. 


RESEARCH METHODOLOGY 

To identifying the studies relevant for our MASEM consisted of 
using the internet search from ThaiLis Digital Collection and the 
electronic theses online system of 71 higher education institutions 
of Thailand. The studies are composed of quantitative research, 
experimental and correlational research, which focus on factors 
relating to the family characteristic, the intellectual characteristic, 
the psychological characteristic, and the classroom environment, 
which affect students' HOTS. The thinking skills consist of three 
factors; analytical thinking, critical thinking, and creative thinking 
published during 1999-2013, which was the period when the Thai 
educational system was renovated and there was more emphasis 
on students’ thinking skills development. Search keywords include 
the following terms: 1) classroom climate, 2) teaching and learning 
methods, 3) teacher behavior, 4) democratic parenting style, 5) 
parental support, 6) attitude toward learning, 7) achievement 
motivation, 8) internal locus of control, 9) intelligence quotient, 10) 
reasoning abilities, 11) analytical thinking 12) critical thinking, 13) 
creative thinking and 14) higher order thinking skills. The search 
initially yielded 300 primary studies from 35 educational institutions 
matching our keywords. The studies were then examined for 
inclusion in our study, using the inclusion criteria. 

Selecting the studies 

Not all the studies were appropriate for inclusion in our analysis. 
Rosenthal (1995) and Wolfswinkel et al. (2013) recommended that 
researchers should assess the quality of the primary studies before 
analyzing the establishing criteria for the inclusion of the primary 
studies by using a multiple-rater technique to evaluate data from 
the primary studies, and assessing inter-rater reliability. Therefore, 
the author processes the research as follows; 

Inclusion criteria 

The studies would be included in the present meta-analysis if it 
satisfied the following inclusion criteria. 1) At least two of the 
constructs included in our hypothetical model were analyzed in the 
studies. 2) The sample in each primary studies are the students of 
the government schools 3) Both bivariate Pearson correlations(r) 
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Table 1. Research Hypotheses and Supporting Literature 

Research hypotheses _ Supporting literature _ 

HI: Classroom environment positively affects Brown and Freeman, 2000; Fleith, 2000; Galton et al., 2009; Chini et al., 

HOTS. _ 2009; Pascarella et al., 2013 _ 

H2: Classroom environment positively affects Ari and Eliassy, 2003; Bong, 2005; Patrick et al., 2007; Nelson and Debacker, 

psychological characteristic of students. _ 2008; Dorman, 2009; Baeten et al., 2013 _ 

H3: Classroom environment positively affects Blumenfeld et al., 1987; Zohar, 1994; Shield and Dockrell, 2008; Barkl et al., 

intellectual characteristic of students. _ 2012; Pascarella et al., 2013 _ 

H4: Family characteristic positively affects Torrance, 1965; Miller and Gerard, 1979; Querido et al., 2002; Lee et al., 

HOTS. _ 2006; Lim and Smith, 2008; Fearon et al., 2013 _ 

H5: Family characteristic positively affects Ginsburg and Bronstein, 1993; Gottfried et al., 1994; Kellan, 2000; Hoang, 

psychological characteristic of students. _ 2007; Umo, 2013 _ 

H6: Family characteristic positively affects Dombusch et al., 1987; McGinn et al., 2005; Houtenville and Conway, 2008; 

intellectual characteristic of students. _ Akinsola, 2011; Wang, 2014 _ 

H7: Psychological characteristic positively affects Richmond and Serna, 1980; Amabile et al., 1990; Moneta and Siu, 2002; 

HOTS. _ Pannells and Claxton, 2008; Lather et al., 2014 _ 

H8: Intellectual characteristic positively affect Mednick and Andrews, 1967; Plucker and Renzulli, 1999; Sternberg and 
HOTS._O’Hara, 1999; Kane et al., 2004; Kim, 2005; Horan, 2007; Silvia, 2008_ 



Figure 1. The theoretical model of the factors affecting higher order thinking skills. 


and sample size were reported in the studies. 4) The sufficient data to 
compute effect sizes according to Glass’ formula were reported in the 
studies. Abstracts of these papers were examined in greater detail. 
After closer inspection of the full papers, only 166 studies from 22 
educational institutions satisfied all the above criteria and were retained 
to create a pooled correlation matrix for the MASEM analysis. 


Intercoder reliability 

The author examined all collected primary studies and recoded 
information on each study’s demographic and substantive features 


to ensure the literature search processed reliability (Cooper and 
Hedges, 1994). 

The studies were coded by 3 authors independently, consisting 
of two research advisers and the author, reaching an intercoder 
agreement of 95%. The level of agreement reached was highly 
satisfactory. Disagreements in coding were resolved through 
discussion for consensus. 


Data analysis 

Two steps of Viswesvaran and Ones (1995) were employed for this 
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MASEM: 

In step 1; the addition of the pooled correlations matrix based on 
Hedges and Olkin (1985) method consisted of three steps; 1.1) 
Transformed correlation coefficients into a standard normal metric 
using Fisher's r-to-Z transformation before calculating a weighted 
average of these transformed scores in fixed-effects model. (Fisher, 
1921; Hedges and Olkin's, 1985) 1.2) Next, we tested the 
homogeneity of correlations from 1.1. Hedges and Olkin’s Q 
statistic was applied to test the homogeneity of the correlations for 
each component. The fixed-effects model is appropriate for 
calculating the pooled correlation matrix when the heterogeneity 
tests are insignificant. Whereas, the random-effects model is proper 
when these tests indicate heterogeneity (Hedges and Vevea, 1998; 
Hunter and Schmidt, 2004). 1.3) After that, we transformed the 
weighted average Fisher's Z-to-r correlation for each pair of all 
variables back to the standard correlational form to the more 
interpretable effects size for reporting. This resulted in a matrix of 
meta-analytic correlations between all variables in the hypothetical 
model. The Comprehensive Meta-Analysis computer program was 
used to perform the data analysis (Borenstein et al., 2009) 

In step 2 of the MASEM, the pooled correlation matrix by the 
true-score population effect sizes of the variable pairs was 
subjected to the SEM technique using the Mplus version7. The 
criteria for assessing the validity of a structural equation model was 
a very good fit with the empirical data from the primary studies 
composed of the Comparative fit index; CFI, the Tucker - Lewis 
index; TLI, the Standardized root mean squared residual; SRMR, 
and the Root mean squared error of approximation; RMSEA. The 
goodness of fit statistics from structural validity shows that very 
good fitting model were CFI and TLI > .95 SRMR and RMSEA < .05 
(Mclachlan and Pell, 2000; Muthen and Muthen, 2009; Byrne, 2012). 
For model sample size, we followed the recommendation of Viswesvaran 
and Ones (1995) to use the harmonic mean as the appropriate sample 
size because it tends to yield the least biased estimates of standard 
errors of parameter estimates. 


RESULTS 

Description of studies 

Studies included in the meta-analysis were highly 
variable in terms of sample sizes that ranged from 411 to 
30,163. The harmonic mean of the sample sizes was 
655. For each effect size, the author used the following 
criteria to assess the effect size magnitudes: small (r< 
0.30), moderate (0.30 < r < 0.50), and large (r > 0.50) 
(Cohen, 1988). 

Among the 78 average weighted correlations obtained 
in the fixed effect model varied from small to large (0.060 
to 0.669); a majority of correlations (40 out of 78) was the 
moderate, 25 correlations was the small, and 13 
correlations was the large. Lower- and upper- bound 
effect sizes for confidence intervals of fixed effect model 
ranged from -0.003-0.683. In the random effect model, 
the effect size varied from small to large (0.060-0.576), a 
majority of correlations (48) was the moderate, 26 
correlations was the small, and 4 correlations was the 
large. Lower- and upper- bound effect sizes for 
confidence intervals of random effect model ranged from 


-0.003-1.958 (show in Appendix A). 


Results of the validity of a structural equation model 
of factors affecting HOTS 

According to the pooled correlation matrix of a structural 
equation model of factors affecting HOTS consisted of 78 
effect sizes in the matrix (show in Appendix B). The result of 
the initial path analysis showed that the model was a very 
good fit with the empirical data from the primary studies with 

;T = 0.035, df = 5, p-value = 1.000, TLI = 1.025, CFI = 
1.000, SRMR = 0.001, RMSEA = 0.000 (Table 2). 

Results of the validity of a structural equation model of 
factors affecting HOTS are shown in Figure 2. 

In accordance with Table 2 and Figure 2, the direction 
of effects is summarized as follows: 

Direct effect factors are as follows. 

The finding showed that three-fourths of the path of 
factors directly affecting HOTS significantly affected 
HOTS. The psychological characteristic (H7: 0.762**) 
indicated a higher effect size than the classroom 
environment (HI: 0.380*) and double in the intellectual 
characteristic (H8: 0.363*). The three latent factors 
explain the variance of 96.8%. However, the family 
characteristic insignificantly affected HOTS. Therefore, 
the study of the contribution of the psychological 
characteristic will enhance students’ HOTS more than the 
classroom environment, and double in the intellectual 
characteristic. If we compare the results with the family 
characteristic, the findings indicated a 7 times higher 
development in students’ HOTS. 

The classroom environment (H2: 0.521**) had 

significant direct effects on the psychological charac¬ 
teristics equal to the family characteristic (H5: 0.414**). 
Therefore, in order to study or research, the development 
of students’ psychological characteristic must focus on 
the enhancement of the classroom environment and the 
family characteristic. Even though the effect size of 
variable in the classroom environment was higher, the 
result indicated that the variable of the psychological 
characteristic must be equally focused on. 

The factors that directly affected the intellectual 
characteristic consisted of the classroom environment 
(H3 : 0.457**), which showed a higher value more than 
twice of the family characteristic (H6 : 0.208*). It can be 
conclude that, study in the contribution of the classroom 
environment will enhance students’ intellectual charac¬ 
teristic more than twice of the family characteristic. 

The four paths of indirect factors affecting HOTS are as 
follows. 

The family characteristic indirectly affected HOTS through 
the psychological characteristics of students (0.315**). 
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Table 2. Direct effects, indirect and total effects of the factors in structural model 


Hypothesis 

Variable 


Direct Effects 

Indirect Effects 

Total Effects 

DV 

IV 

Est. 

S.E. 

Z 

Est. 

S.E. 

Z 

Est. 

S.E. 

Z 

H 1 

HOTS 

CIEnv 

0.380* 

0.179 

2.125 


- 

- 




- 

HOTS 

CIEnv to IntCh 

- 

- 

- 

0.166** 

0.080 

2.079 

0.943** 

0.139 

6.804 

- 

HOTS 

CIEnv to PsyCh 

- 

- 

- 

0.397** 

0.142 

2.803 




H 4 

HOTS 

FaCh 

-0.275 

0.237 

-1.161 

- 

- 

- 




- 

HOTS 

FaCh to IntCh 

- 

- 

- 

0.076 

0.053 

1.425 

0.115 

0.141 

0.817 

- 

HOTS 

FaCh to PsyCh 

- 

- 

- 

0.315* 

0.155 

2.037 




H 7 

HOTS 

PsyCh 

0.762** 

0.288 

2.650 

- 

- 

- 

0.762** 

0.288 

2.650 

H 8 

HOTS 

IntCh 

0.363* 

0.160 

2.271 

- 

- 

- 

0.363* 

0.160 

2.271 

H 2 

PsyCh 

CIEnv 

0.521** 

0.084 

6.220 

- 

- 

- 

0.521** 

0.084 

6.220 

H 5 

PsyCh 

FaCh 

0.414** 

0.089 

4.627 

- 

- 

- 

0.414** 

0.089 

4.627 

H 3 

IntCh 

CIEnv 

0.457** 

0.082 

5.567 

- 

- 

- 

0.457** 

0.082 

5.567 

H 6 

IntCh 

FaCh 

0.208* 

0.098 

2.115 

- 

- 

- 

0.208* 

0.098 

2.115 


Notes: X 2 = 0.035, df = 5, % 2 /df = 0.007, p-value = 1.000, CFI = 1.000, TLI = 1.025, SRMR = O.OOf , RMSEA = 0.000, R 2 (HOTS) = 0.968, *p < .05, 

**p< .01. 


0.65S~ 04HT 



0 . 600 ** 03 « 9 ** 0320 ** 0363 ** 0495 ** 


Figure 2. Results of the MASEM for a structural equation model of factors affecting HOTS. 


Example of the value of indirect effects (0.414x0.762 = 
0.315) was developed by two paths. 1) The family 
characteristic had a direct effect on the psychological 


characteristic of students (H4: 0.414**). (2) The psycho¬ 
logical characteristics of students directly affected HOTS 
(H7:0.762**). In conclusion, the family characteristic 
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affects the improvement of the psychological charac¬ 
teristics and will enhance students’ HOTS. 

The family characteristic had insignificant indirect effects 
on the intellectual characteristic. The study indicates that 
study or research in the contribution of the family 
characteristic affects the improvement of the intellectual 
characteristic will not enhance students’ HOTS. 

The classroom environment indirectly affected HOTS through 
the psychological characteristics of students (0.397**). It is 
concluded that the classroom environment positively 
affects the psychological characteristics and also 
increases students’ HOTS. 

The classroom environment indirectly affected HOTS through 
the intellectual characteristic (0.166**). In conclusion, the 
study shows that the classroom environment positively 
affects the intellectual characteristic and enhances 
students’ HOTS. 


DISCUSSION 

This study brings conceptual and empirical clarity to the 
factors affecting HOTS based on the MASEM method. Our 
study makes four major contributions to theory, as follows. 

The psychological characteristic, the classroom environ¬ 
ment, and the intellectual characteristic of students 
directly affect HOTS, which supports the hypothesis. 
However, the family characteristic insignificantly affects 
HOTS. The result may be caused by the development of 
the hypothesis, which determined only direct effects. 
When this path included in the model had a variety 
complex variables, it did not support the hypothesis (Ali 
and Hamed, 2015). The psychological characteristic had 
effects on HOTS more than any other. This may be 
explained because these variables can be continuously 
developed by various techniques of the learning process, 
the classroom environment, and parents’ support (Hoang, 
2007; Dorman, 2009; Baeten et al., 2013; Umo, 2013). 

The classroom environment and the family characteristic 
directly affect the intellectual characteristic, which supports 
the hypothesis. Morris and Maisto (2002) assert that the 
elements of social environment affect the intellectual 
characteristic. The classroom environment and the family 
condition are considered to be parts of the social 
environment. Additionally, this study found that the 
classroom environment has more than twice an effect of the 
family characteristic. This may result from the classroom 
environment and learning management that aim to 
encourage students to learn and develop their intellectual 
characteristic. Moreover, classroom management can 
design situations or experiments practice students thinking 
skills in various ways during the period of learning (School 
Drug Education and Road Aware, 2013). 

The psychological characteristic of students is an 
important mediator variable for HOTS. The study indicates 
two indirect effects through students’ psychological 


characteristic: the classroom environment and the family 
characteristic. The results may be caused by the attribute 
of SEM analysis that able to analyze various effects 
including direct effect, moderating effect, and reverse 
effect. These also allow to identify linear and additive 
relationships of recursive and non- recursive model, as 
well as indirect effect through the mediator variable 
(Schumacker and Lomax, 2010; Barbara, 2012). The 
result of this research shows that the classroom 
environment has more an indirect effect than the family 
characteristic. The results may be caused by the effective 
instructional management, which benefits in organizing 
classroom environment to support the feelings, attitudes, 
knowledge, and thinking skills of students (Nelson and 
Debacker, 2008; Chini et al., 2009; Pascarella et al., 2013). 

The family characteristic had insignificant indirect 
effects on the intellectual characteristic. The results may 
because the intellectual characteristic had several effects 
on both the classroom environment and the family 
characteristic. When this path included the complex 
model, it did not support the hypothesis. However, the 
family characteristic is also an important factor that is 
indirectly affected through the psychological characteristic 
and increases students’ HOTS. 

Conclusion 

Within the organizational literature, the study of factors 
affecting HOTS has been conducted for many years. 
Researchers have chosen to study a variety of variables 
and proposed a variety of models based on their 
individual interests, and there is no systematic integration 
among them. Moreover, some research findings are 
inconsistent with other studies and have become difficult 
to draw conclusions from the literature reviews. To solve 
these problems, this study collects the variables affecting 
HOTS to synthesize and find the conclusion for MASEM. 
This research contributes systematic integration among 
the variables. The research findings confirm the 
concepts, theories and importance of factors based on 
the structural equation model of factors affecting HOTS. 
The model is systematically designed from various 
concepts and theories of HOTS, which makes powerful 
and strong results and gains a boarder conclusion than 
the conclusions of one single primary study (Hunter and 
Schmidt, 2004). Moreover, this study extends the concept 
of research synthesis by using advanced statistics to 
behavioral and social sciences study. 

Suggestions 

On the basis of the results of this study, we have several 
suggestions for future research and practical applications; 

1. The findings indicate that the classroom climate, 
teaching and learning methods affect HOTS of students. 
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Additionally, the psychological factor should be consi¬ 
dered and applied to the classroom environment. For 
example, with the positive learning activity management, 
the classroom climate should support positive thinking in 
learning, teaching behavior or personalities of teachers to 
support an attitude toward learning. Techniques to moti¬ 
vate students to learn and express ideas can enhance 
HOTS of student as well. 

2. The democratic parenting style and support of the 
family will help students improve their attitudes towards 
learning, achievement motivation, and self-trust, which 
will affect HOTS. Therefore, the parents should take care 
of their children closely and fairly, and provide students 
with an opportunity to share their ideas, make decisions, 
and solve problems. Additionally, the parents should 
encourage their children to participate in activities in and 
out of the classroom. 

3. There are many studies on social science, which 
contain various models and variables. It is possible to 
synthesize those variables and make conclusions. This 
strategy extends the limit of the study of some variables 
in social science study. However, even this research got 
the conclusion. Future research may apply this 
information for research and development in practice 
such as development of learning strategy for contributing 
to students’ HOTS. It is possible to apply the MASEM, 
which will not only develop students’ HOTS, but also to 
extend the area of the MASEM study. Additionally, this 
form of learning management, from research synthesis 
with the advance statistical method, will add more value 
to future research. 

4. The results of Q statistics indicated that the hetero¬ 
geneity among effect size across studies (Hedges and 
Olkin, 1985). Therefore, future research should 
investigate the sources of this heterogeneity through 
moderator analyses, and the limitation of the process 
must be considered on the possibility of sufficient (Lipsey, 
1994; Card, 2012). 
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Appendix A. Fixed-effect and Random-effect model of correlations between the variables in the model of 
factors affecting higher order thinking skills 


r 

k 

N 

Fixed-Effect Model 

Random-Effect Model 

Q(df) 

Weight r 

Cl 95% 

Weight r 

Cl 95% 

TeM- CICIi 

15 

13641 

0.550 

0.538-0.561 

0.487 

0.401 - 0.564 

533.815(14) 

TeB - CICIi 

20 

17387 

0.380 

0.367- 0.393 

0.390 

0.265 - 0.501 

1635.932(19) 

TeB - TeM 

1 

880 

0.381 

0.344 - 0.455 

0.401 

0.323 - 0.436 

0.000 

Sup - CICIi 

8 

5161 

0.476 

0.454 - 0.497 

0.410 

0.246 - 0.551 

311.612(7) 

Sup - TeM 

2 

1123 

0.540 

0.498 - 0.580 

0.405 

-0.152-0.767 

85.987(1) 

Sup - TeB 

9 

5073 

0.405 

0.383 - 0.428 

0.371 

0.188-0.53 

416.133 (8) 

Dmo - CICIi 

17 

12315 

0.273 

0.256-0.289 

0.272 

0.172-0.366 

552.270(16) 

Dmo - TeM 

12 

9273 

0.380 

0.360 - 0.400 

0.358 

0.246 - 0.461 

378.57(11) 

Dmo - TeB 

9 

5471 

0.434 

0.413-0.456 

0.385 

0.159-0.573 

673.675 (8) 

Dmo - Sup 

1 

411 

0.520 

0.507-0.637 

0.576 

0.445 - 0.587 

0.000 

Atti- CICIi 

25 

18008 

0.411 

0.399 - 0.423 

0.376 

0.301 - 0.445 

782.346 (24) 

Atti- TeM 

9 

6770 

0.351 

0.330 - 0.372 

0.356 

0.267 - 0.439 

131.845 (8) 

Atti- TeB 

17 

10368 

0.437 

0.420 - 0.452 

0.424 

0.308 - 0.527 

759.667(16) 

Atti- Sup 

15 

13359 

0.414 

0.399 - 0.427 

0.39 

0.277 - 0.493 

769.592(14) 

Atti- Dmo 

16 

15442 

0.309 

0.294-0.323 

0.328 

0.239-0.412 

550.770(15) 

Moti - CICIi 

23 

16352 

0.405 

0.393-0.419 

0.374 

0.272 - 0.468 

1191.321 (22) 

Moti - TeM 

10 

7293 

0.374 

0.354-0.394 

0.380 

0.271 - 0.479 

246.024 (9) 

Moti - TeB 

11 

7076 

0.428 

0.409 - 0.448 

0.402 

0.294 - 0.501 

278.029(10) 

Moti - Sup 

9 

6398 

0.468 

0.448 - 0.486 

0.416 

0.311 -0.511 

194.380 (8) 

Moti - Dmo 

18 

15998 

0.343 

0.328- 0.356 

0.314 

0.237 - 0.388 

464.839(17) 

Moti - Atti 

37 

30163 

0.444 

0.435 - 0.453 

0.419 

0.36 - 0.476 

1362.962 (36) 

Loc - CICIi 

17 

12101 

0.399 

0.384-0.414 

0.383 

0.292 - 0.465 

506.883(16) 

Loc - TeM 

7 

5095 

0.304 

0.279 - 0.329 

0.306 

0.174-0.427 

146.969 (6) 

Loc - TeB 

11 

6797 

0.548 

0.531 - 0.565 

0.436 

0.177-0.638 

1414.000(10) 

Loc - Sup 

7 

5963 

0.541 

0.523 - 0.559 

0.426 

0.25 - 0.574 

358.098 (6) 

Loc - Dmo 

16 

13886 

0.380 

0.366 - 0.394 

0.364 

0.286 - 0.437 

390.421 (15) 

Loc - Atti 

19 

17857 

0.373 

0.360 - 0.385 

0.354 

0.262 - 0.439 

840.497(18) 

Loc - Moti 

19 

17495 

0.417 

0.405 - 0.429 

0.404 

0.297 - 0.5 

1164.873(18) 

IQ - CICIi 

7 

3738 

0.212 

0.181 - 0.242 

0.239 

0.016-0.439 

285.942 (6) 

IQ - TeM 

1 

810 

0.153 

0.086-0.220 

0.154 

0.085-0.219 

0.000 

IQ -TeB 

1 

528 

0.453 

0.420 - 0.550 

0.488 

0.382-0.518 

0.000 

IQ - Sup 

1 

971 

0.060 

-0.003 - 0.122 

0.060 

-0.003-0.122 

0.000 

IQ - Dmo 

6 

5125 

0.194 

0.167-0.220 

0.150 

-0.039 - 0.329 

224.824 (5) 

IQ - Atti 

12 

9656 

0.267 

0.249 - 0.286 

0.291 

0.292 - 0.409 

469.773(11) 

IQ - Moti 

14 

10434 

0.180 

0.162-0.198 

0.217 

0.111 -0.319 

408.576(13) 

IQ - Loc 

7 

4759 

0.397 

0.372 - 0.420 

0.377 

0.148-0.568 

438.936(6) 

Reas - CICIi 

14 

9169 

0.193 

0.172-0.212 

0.183 

0.092 - 0.27 

256.402(13) 

Reas - TeM 

7 

3858 

0.248 

0.217-0.278 

0.252 

0.064 - 0.423 

207.985 (6) 

Reas - TeB 

13 

8576 

0.424 

0.406 - 0.441 

0.338 

0.135-0.515 

1225.289 (12) 

Reas - Sup 

6 

7985 

0.278 

0.258-0.299 

0.260 

-0.063 - 0.533 

1072.127 (5) 

Reas - Dmo 

8 

7421 

0.239 

0.217-0.261 

0.226 

0.108-0.338 

168.861 (7) 

Reas - Atti 

18 

17454 

0.294 

0.280 - 0.308 

0.290 

0.138-0.429 

1886.054(17) 

Reas - Moti 

10 

9339 

0.210 

0.191 -0.230 

0.230 

0.104-0.348 

327.748 (9) 

Reas -Loc 

14 

12565 

0.478 

0.464 - 0.491 

0.399 

0.212-0.558 

1713.229 (13) 

Reas - IQ 

3 

1701 

0.205 

0.160-0.251 

0.272 

-0.074 - 0.559 

99.955 (2) 

AnT- CICIi 

17 

11781 

0.258 

0.241 - 0.275 

0.251 

0.140-0.354 

625.589(16) 

AnT- TeM 

25 

3856 

0.437 

0.411 - 0.463 

0.546 

0.425 - 0.649 

489.619(24 
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AnT- TeB 

12 

7993 

0.505 

0.488-0.521 

0.426 

-0.33 - 0.663 

2608.866(11) 

AnT- Sup 

3 

1790 

0.279 

0.235- 0.321 

0.252 

-0.03 - 0.498 

73.060 (2) 

AnT- Dmo 

9 

5637 

0.223 

0.198-0.248 

0.244 

0.067 - 0.405 

377.386 (8) 

AnT- Atti 

27 

17975 

0.327 

0.314-0.340 

0.327 

0.242 - 0.408 

1026.142 (26) 

AnT- Moti 

24 

15865 

0.304 

0.289 - 0.318 

0.301 

0.217-0.381 

756.259 (23) 

AnT- Loc 

13 

8496 

0.404 

0.386 - 0.422 

0.377 

0.139-0.573 

1710.799 (12) 

AnT- IQ 

9 

6428 

0.550 

0.532-0.566 

0.506 

0.378-0.616 

333.424 (8) 

AnT- Reas 

11 

6712 

0.392 

0.371 - 0.412 

0.358 

0.115-0.561 

1140.852(10) 

CriT- CICIi 

16 

12183 

0.244 

0.227-0.261 

0.219 

0.141 - 0.296 

303.052(15) 

CriT- TeM 

24 

6527 

0.435 

0.414-0.454 

0.466 

0.338 - 0.577 

701.392 (23) 

CriT-TeB 

13 

8088 

0.584 

0.569 - 0.598 

0.306 

-0.059 - 0.599 

3397.708 (12) 

CriT- Sup 

5 

4508 

0.521 

0.499 - 0.542 

0.341 

-0.109-0.675 

947.090 (4) 

CriT- Dmo 

17 

13234 

0.304 

0.289 - 0.319 

0.250 

0.152-0.343 

544.581 (16) 

CriT- Atti 

12 

9338 

0.225 

0.206 - 0.244 

0.204 

0.117-0.288 

206.900(11) 

CriT- Moti 

12 

9388 

0.23 

0.211 - 0.249 

0.235 

0.111 -0.352 

428.627(11) 

CriT- Loc 

19 

14570 

0.495 

0.482 - 0.506 

0.396 

0.232 - 0.538 

2213.088(18) 

CriT- IQ 

4 

3410 

0.658 

0.639 - 0.677 

0.499 

0.188- 1.958 

326.943 (3) 

CriT- Reas 

14 

8730 

0.573 

0.558 - 0.587 

0.431 

0.254 - 0.580 

1143.277 (13) 

CriT- AnT 

8 

4820 

0.483 

0.461 - 0.504 

0.382 

0.110-0.601 

628.668 (7) 

CreT- CiCIi 

10 

6068 

0.669 

0.655-0.683 

0.394 

-0.134-0.747 

4187.565 (9) 

CreT- TeM 

28 

4087 

0.635 

0.616-0.653 

0.539 

0.409 - 0.648 

654.507 (27) 

CreT- TeB 

7 

3723 

0.439 

0.413-0.465 

0.410 

0.175-0.601 

382.131 (6) 

CreT- Sup 

9 

8428 

0.164 

0.142-0.184 

0.210 

0.111 -0.305 

172.476 (8) 

CreT- Dmo 

4 

1908 

0.228 

0.185-0.270 

0.230 

-0.071 - 0.493 

133.339 (3) 

CreT - Atti 

12 

9607 

0.240 

0.221 - 0.259 

0.297 

0.171 - 0.412 

457.058(11) 

CreT- Moti 

8 

3766 

0.260 

0.230 - 0.289 

0.263 

0.036 - 0.464 

364.394 (7) 

CreT- Loc 

3 

2056 

0.273 

0.233 - 0.313 

0.322 

0.007 - 0.580 

107.155 (2) 

CreT- IQ 

7 

2934 

0.372 

0.341 - 0.403 

0.297 

0.019-0.531 

358.273 (6) 

CreT- Reas 

5 

5764 

0.402 

0.380 - 0.424 

0.369 

0.245 - 0.481 

97.194 (4) 

CreT- AnT 

4 

2015 

0.427 

0.390 - 0.462 

0.319 

-0.085 - 0.633 

416.659 (3) 

CreT- CriT 

4 

1910 

0.498 

0.464-0.531 

0.380 

-0.069 - 0.701 

309.952 (3) 


Notes: r = Correlation between the variables: k ^Number of studies; N=Number of observations; Weight r= 
Weighted mean effect size; Cl= Confidence intervals; Q = Chi-square test; highlight = Weighted mean effect 
size applied to the correlation matrix; CiCIi = Classroom climate; TeM = Teaching and learning methods; 
TeB = Teacher behavior; Dmo = Democratic parenting style; Sup = Parental support; Atti = Attitude toward 
learning; Moti = Achievement motivation; Loc = Internal locus of control; IQ = Intelligence quotient; Reas = 
Reasoning abilities; AnT = Analytical thinking: CriT = Critical Thinking; CreT = Creative Thinking 
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Appendix B. Meta-analytic correlation matrix in the model of factors affecting higher order thinking skills 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

1. CICIi 

- 














0.487(r) 













2. Lear 

15 

- 













13641 













3. TeB 

0.390(r) 20 
17387 

0.381(f) 

1 

880 

- 












0.410(r) 

0.405(r) 

0.371 (r) 











4. Sup 

8 

2 

9 

- 











5161 

1123 

5073 












0.272(r) 

0.358(r) 

0.385(r) 

0.52(f) 










5. Dmo 

17 

12 

9 

1 

- 










12315 

9273 

5471 

411 











0.376(r) 

0.356(r) 

0.424(r) 

0.39(r) 

0.328(r) 









6. Atti 

25 

9 

17 

15 

16 

- 









18008 

6770 

10368 

13359 

15442 










0.374(r) 

0.38(r) 

0.402(r) 

0.416(r) 

0.314(r) 

0.419(r) 








7. Moti 

23 

10 

11 

9 

18 

37 

- 








16352 

7293 

7076 

6398 

15998 

30163 









0.383(r) 

0.306(r) 

0.436(r) 

0.426(r) 

0.364(r) 

0.354(r) 

0.404(r) 







8. Loc 

17 

7 

11 

7 

16 

19 

19 

- 







12101 

5095 

6797 

5963 

13886 

17857 

17495 








0.239(r) 

0.153(f) 

0.453(f) 

0.060(f) 

0.150(r) 

0.291 (r) 

0.217 (r) 

0.377(r) 






9. IQ 

7 

1 

1 

1 

6 

12 

14 

7 

- 






3738 

810 

528 

971 

5125 

9656 

10434 

4759 







0.183(r) 

0.252(r) 

0.338(r) 

0.260(r) 

0.226(r) 

0.290(r) 

0.230(r) 

0.399(r) 

0.272(r) 





10. Reas 

14 

7 

13 

6 

8 

18 

10 

14 

3 

- 





9169 

3858 

8576 

7985 

7421 

17454 

9339 

12565 

1701 






0.251 (r) 

0.546(r) 

0.426(r) 

0.252(r) 

0.244(r) 

0.327 (r) 

0.301 (r) 

0.377(r) 

0.506(r) 

0.358(r) 




11.AnT 

17 

25 

12 

3 

9 

27 

24 

13 

9 

11 

- 




11781 

3856 

7993 

1790 

5637 

17975 

15865 

8496 

6428 

6712 





0.219(r) 

0.466(r) 

0.306(r) 

0.341 (r) 

0.250(r) 

0.204(r) 

0.235(r) 

0.396(r) 

0.499(r) 

0.431 (r) 

0.382(r) 



12. CriT 

16 

24 

13 

5 

17 

12 

12 

19 

4 

14 

8 

- 



12183 

6527 

8088 

4508 

13234 

9338 

9388 

14570 

3410 

8730 

4820 




0.394(r) 

0.539(r) 

0.410(r) 

0.210(r) 

0.230(r) 

0.297 (r) 

0.263(r) 

0.322(r) 

0.297(r) 

0.369(r) 

0.319(r) 

0.380(r) 


13.CreT 

10 

28 

7 

9 

4 

12 

8 

3 

7 

5 

4 

4 

- 


6068 

4087 

3723 

8428 

1908 

9607 

3766 

2056 

2934 

5764 

2015 

1910 



Notes: upper row = Weighted mean effect size; middle row = Number of independent correlation matrices obtained for each construct ; lower row =Total sample size obtained for 
each construct; (f) = Weighted mean effect size based on fixed-effects model, (r) = Weighted mean effect size based on random-effects model. 



